[sandbox audit] Pin, verify and attest-ready the published server binary - #24

[sandbox audit] — PR 12 of 13 in this repo's stack; merge in order.
Previous: #26 · Next: #27
Review only the commits this PR adds on top of its base; bases collapse to main as the stack lands.

Companion client PR: huggingface/huggingface_hub#4837 — these two have to land together, and the client cannot merge until this workflow has published once (see "Rollout" below).

What was wrong

Every sandbox job downloads sbx-server from a public bucket, chmod +xes it and runs it as root, as PID 1, holding the derived sandbox token. Two things about that pipeline were weaker than they needed to be.

The object every client consumes is mutable, and the pipeline that writes it was loose. publish.yml published an immutable sbx-server-<commit> copy — good — and then overwrote the sbx-server alias that every client actually fetches. Around that:

• on: workflow_dispatch against ref: main, so the published bytes were whatever main happened to say at dispatch time — not a named, reviewed commit.
• actions/checkout@v7, a mutable tag.
• curl -LsSf https://hf.co/cli/install.sh | bash, in the job holding the bucket-write OIDC token. That is the sharpest edge in the file: an unpinned remote script running with publish rights over every sandbox of every user.
• cargo build --release with no --locked, so dependency versions were re-resolved at publish time rather than taken from the reviewed Cargo.lock.
• No environment gate on the alias update.

Nothing suggests any of this was exploited — the published binary matches this source. The finding is the absence of a guarantee about the next fetch, and the blast radius: a single write to that bucket path reaches every sandbox of every user.

/health reported version, and nothing negotiated it. version moves for a doc fix as readily as for a protocol break, so a client cannot decide from it whether talking to a given server is safe. That matters because a pool host keeps the binary it downloaded at boot for up to 24h: a client and a server from different releases meet in production routinely. The per-sandbox capability token from #18 is exactly the kind of change that fails quietly under that — as a 403 on an unrelated route, several calls later.

What changed

Protocol negotiation. /health now reports protocol, an integer, currently 2 (1 was the pre-per-sandbox-token contract). It moves only when the wire contract changes in a way a client can be wrong about — never for an additive one. The payload moved into its own health() so the handshake can be asserted on without a socket.

Content-addressed publishing. The workflow now publishes, per build:

The last two are remote copies of the digest object, so every name in the bucket is byte-identical to the digest by construction, and existing clients keep working unchanged. The digest is printed in the run summary in a form that can be pasted straight into the client.

Workflow hardening.

• Triggers on a v* tag, so the published bytes always correspond to a named, reviewed commit. workflow_dispatch stays as an emergency path but now requires a ref input — neither trigger builds "whatever main says right now".
• actions/* pinned by commit SHA with Dependabot's # vX.Y.Z comment convention, so updates stay automated.
• The hf CLI is installed from PyPI at a pinned version into a venv, instead of piping a remote script into bash.
• cargo build --locked.
• Build and publish are separate jobs. The build job has no id-token permission and no environment, so nothing running during compilation — a build script, a proc macro, a dependency — shares a job with the bucket-write credential. The digest is recomputed from the bytes the publish job is about to upload rather than trusted across the job boundary. This is the achievable form of the "build with no network beyond the crates registry" idea; real egress control would need a third-party action, which felt like the wrong thing to add to a supply-chain PR.
• environment: production, so the alias update can be gated on reviewers and the bucket-write OIDC scope can be narrowed to it.

README gains a "Releasing" section. Pinning the digest in the client couples the two repos — every server release now needs a client release — and that trade needs to be written down rather than learned.

Validation

cargo test:

cargo test: 39 passed (1 suite, 0.00s)

The new case asserts /health reports protocol as an integer equal to PROTOCOL. Dropping or renaming that field would turn the client's check into a silent no-op, which is the exact failure it exists to prevent.

cargo build --locked --release --target x86_64-unknown-linux-musl succeeds, and the resulting binary runs:

sbx-server 0.6.0 listening on 0.0.0.0:49912 (mode: dedicated, auth: required, landlock: abi 4 [fs,refer,truncate,no_tcp_bind])
sbx-server: idle for 5000ms, shutting down

actionlint 1.7.12 on publish.yml: clean, exit 0.

The bootstrap that consumes all of this was exercised end to end against a local python3 -m http.server across 14 image/tool shapes — see the client PR for the table.

Not validated

• The workflow has not been run. It needs a dry run on a branch with the alias-update step disabled before the first real publish, and it will not work until the production environment exists and the bucket-write OIDC scope is pointed at it.
• No live HF Jobs run. The digest path only really exists inside a job, so one Sandbox.create() under a namespace we control is still owed — that lives on the client PR.

Rollout

The two repos are now coupled, in this order:

1. Merge this PR.
2. Tag v0.6.1 (or whatever the release is) and let the workflow publish. Grab the digest from the run summary.
3. Put that digest in SANDBOX_SERVER_SHA256 on the client PR, then merge it.

Between 1 and 3 nothing changes for users: the alias keeps being updated and no client pins a digest yet. The client PR currently pins a local musl build of this branch, which does not exist in the bucket — so it must not merge before step 2.

Not in this PR

Signing and attestation. An actions/attest-build-provenance (or cosign) signature over the digest, and client-side signature verification, is the natural next step. The pinned digest is what actually closes the gap for now, because it is baked into a reviewed client release; a signature adds provenance tied to the commit, and needs verification tooling available inside arbitrary sandbox images to be worth much. Follow-up.

Crate version. I left Cargo.toml at 0.6.0. Adding a field to /health is additive and PROTOCOL stays 2 (the contract #18 introduced), so nothing incompatible happened here — but if the team's convention is to bump on any wire change, say so and I will.

Adjacent, not touched: ci.yml still uses mutable action tags (actions/checkout@v7, actions/cache@v4; #13 is bumping the latter) and still builds the gnu target while this workflow ships musl. Both belong to other PRs in this series.